Detecting malicious activity on an endpoint based on real-time system events

ABSTRACT

Techniques for detecting malicious activity on an endpoint based on real-time system events are disclosed. In some embodiments, a system/process/computer program product for detecting malicious activity on an endpoint based on real-time system events includes monitoring an endpoint for malicious activity using an endpoint agent, in which the endpoint comprises a local device; detecting malicious activity associated with an application on the endpoint based on real-time system events using the endpoint agent based on a set of rules; and in response to detecting malicious activity on the endpoint based on real-time system events using the endpoint agent, performing a security response based on a security policy.

BACKGROUND OF THE INVENTION

Malware is a general term commonly used to refer to malicious software(e.g., including a variety of hostile, intrusive, and/or otherwiseunwanted software). Malware can be in the form of code, scripts, activecontent, and/or other software. Example uses of malware includedisrupting computer and/or network operations, stealing proprietaryinformation (e.g., confidential information, such as identity,financial, and/or intellectual property related information), and/orgaining access to private/proprietary computer systems and/or computernetworks. Unfortunately, as techniques are developed to help detect andmitigate malware, nefarious authors find ways to circumvent suchefforts. Accordingly, there is an ongoing need for improvements totechniques for identifying and mitigating malware.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 illustrates an example of an environment in which malware aredetected and prevented from causing harm in accordance with someembodiments.

FIG. 2 illustrates a data appliance in accordance with some embodiments.

FIG. 3 illustrates an example of logical components included in a systemfor performing static and dynamic analysis on samples in accordance withsome embodiments.

FIG. 4 illustrates an architecture of a Behavior Threat Protection (BTP)engine for detecting malicious activity on an endpoint based onreal-time events in accordance with some embodiments.

FIG. 5 illustrates an architecture flow of a Behavior Threat Protection(BTP) engine for detecting malicious activity on an endpoint based onreal-time events in accordance with some embodiments.

FIG. 6 illustrates a process for normalizing an event to a CLIPS Fact inaccordance with some embodiments.

FIG. 7 is a screen diagram illustrating a behavioral threat protectionanalysis from a security agent console of an endpoint agent inaccordance with some embodiments.

FIG. 8 is a flow diagram illustrating a process for detecting maliciousactivity on an endpoint based on real-time events in accordance withsome embodiments.

FIG. 9 is another flow diagram illustrating a process for detectingmalicious activity on an endpoint based on real-time events inaccordance with some embodiments.

FIG. 10 is another flow diagram illustrating a process for detectingmalicious activity on an endpoint based on real-time events inaccordance with some embodiments.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

Overview of Detecting Malicious Activity on an Endpoint Based onReal-Time System Events

Malware as used herein generally refers to malicious software (e.g.,including a variety of hostile, intrusive, and/or otherwise unwantedsoftware). Malware can be in the form of code, scripts, active content,and/or other software. Example uses of malware include disruptingcomputer and/or network operations, stealing proprietary information(e.g., confidential information, such as identity, financial, and/orintellectual property related information), and/or gaining access toprivate/proprietary computer systems and/or computer networks. Maliciousactivity as used herein generally refers to activity associated withmalware and/or other forms of anomalous activity.

Various endpoint security solutions generally exist. For example, anendpoint agent (e.g., security software) executed on an endpoint (e.g.,endpoint device) is often provided for protecting the endpoint.

However, malware is constantly changing and evolving, and to protectfrom such malware, the endpoint agent generally has to be adapted and/orupdated to be able to detect new forms of malware. As such, the endpointagent generally should be sufficiently flexible to be able to be updatedin a way that allows for new detection logic (e.g., new detection rules)to be timely and efficiently distributed to millions of protectedendpoint devices (e.g., using a push and/or pull distribution mechanismto deploy such updates). In addition, updating the detection logic istypically performed by security analysts (e.g., a set of malwareresearchers, such as in a research group for a commercial securityservice provider) within a vendor company for the endpoint securitysolution, which can limit the capacity to quickly implement and deploynew security capabilities (e.g., to craft the new detection rules fornew malware or other security threats).

Many of today's endpoint security solutions utilize code updates and newagent versions to deliver new security content. As a result, the typicaltime from identifying a security gap (e.g., malware or other securitythreats) to customer deployment of the code updates and new agentversion to address the security gap can be on the order of four to sixmonths.

As such, more timely and efficient techniques for deploying new securitycontent for endpoint security solutions are desired. For example, it isdesired to provide new techniques to provide timely and efficientupdates to the endpoint agents to facilitate having the endpoint agentsbeing able to prevent a new attack once they detect the attack once.Specifically, it is desired to provide a mechanism to provide newsecurity content quickly to endpoint agents deployed by customers,without necessarily requiring binary updates (e.g., code updates) whichgenerally may force recertification and upgrades from the customers.

Accordingly, new and improved techniques for detecting maliciousactivity on an endpoint based on real-time system events are disclosedas will be further described below. For example, to expand malwareprotection capabilities on endpoints, a new Behavioral Threat Protection(BTP) engine (e.g., a BTP component/module, which can be implemented ina high-level programming language, such as C++, Go, Java, or anotherhigh-level programming language, and is cross-platform, such as on theMicrosoft Windows® operating systems, Apple MacOS® operating systems,Linux operating systems, etc.) is provided that provides a mechanism fordetecting malicious activity on an endpoint based on real-time systemevents and which can be efficiently and timely updated using contentupdates (e.g., new detection rules) to detect new forms/types ofmalicious activity (e.g., without necessarily requiring binary/codeupdates).

In some embodiments, a system/process/computer program product fordetecting malicious activity on an endpoint based on real-time systemevents includes monitoring an endpoint for malicious activity using anendpoint agent, in which the endpoint comprises a local device;detecting malicious activity associated with an application on theendpoint based on real-time system events using the endpoint agent basedon a set of rules; and in response to detecting malicious activity onthe endpoint based on real-time system events using the endpoint agent,performing a security response based on a security policy.

For example, the disclosed techniques for detecting malicious activityon an endpoint based on real-time system events can detect an attempt bythe application to take an action that would violate the set of rules,in which the set of rules includes one or more updated detection rules.

As another example, the disclosed techniques for detecting maliciousactivity on an endpoint based on real-time system events can beconfigured to alert when a user connects to file servers that the userhas not previously connected to based on an observed pattern of theuser's network related activities (e.g., if the endpoint security agentstarts to detect one or more host actions performed by an authenticateduser that was not previously observed on this host machine, then theendpoint security agent can be configured to perform a responsiveaction, such as to log, alert, block, or perform some other responsiveaction(s)).

These and other for techniques for detecting malicious activity on anendpoint based on real-time system events will be further describedbelow with respect to various embodiments.

Example System Environments for Detecting Malicious Activity on anEndpoint Based on Real-Time System Events

FIG. 1 illustrates an example of an environment in which malware (e.g.,malware content, malware applications, and/or other forms of malware)are detected and prevented from causing harm in accordance with someembodiments. The term “application” is used herein to generally refer toprograms, bundles of programs, manifests, packages, etc., irrespectiveof form/platform. The term “malware” as used herein generally refers toan application and/or other dynamic/executable content that engages inbehaviors, whether clandestinely or not (and, whether illegal or not),of which a user (or as applicable, an administrator or other entityspecifying user device policies) does not approve/would not approve iffully informed. Examples of malware include viruses, rootkits, spyware,key loggers, etc.

Various example forms of malware exist including various forms ofapplications and/or other dynamic/executable content that engages in theabove-described behaviors, whether clandestinely or not (and, whetherillegal or not), of which a user (or as applicable, an administrator orother entity specifying user device policies) does not approve/would notapprove if fully informed. One particular example of mobile malware(e.g., mobile app malware) is a malicious .apk file that appears to anend user to be a free game, but stealthily sends SMS premium messages(e.g., costing $10 each), running up the end user's phone bill. Anotherexample of mobile malware is an application that stealthily collects theuser's contacts and sends them to a spammer. Yet another example ofmobile malware is an application that collects and reports to a remoteserver the end user's location (but does not offer a location basedservice to the user, such as a legitimate mapping service wouldprovide). Other forms of malware can also be detected/thwarted using thetechniques described herein. Further, while various techniques are, insome cases, described herein in conjunction with use mobile devices(e.g., cellular telephones/tablet devices), use of techniques describedherein is not limited to the context of any particular form/type of suchendpoint devices. For example, techniques described herein can also beused in conjunction with applications used by other types of devices,such as laptop/desktop computers, gaming consoles, set-top boxes,Internet of Things (IoT) devices, etc. As one example, an end usercomputer (e.g., at a home, in a school, in an enterprise environment,etc.) can make use of the techniques described herein.

Suppose a nefarious individual wishes to propagate malware (such asmalware 130) via system 120 to end users. A variety of approaches can beused by the nefarious individual. As one example, the individual canupload malware 130 to a software distribution platform such as platform134 (also referred to as an “an app store”). The nefarious individualhopes that unsuspecting users of platform 134 (e.g., any of applicableclient devices 104, 106, 107, and 108) will download the maliciousapplication/content 130 from platform 134 and install it on theirdevices. Example embodiments of platform 134 include Google Play, theiOS App Store, BlackBerry World, the Windows Phone Store, the MicrosoftStore, and the Amazon Appstore. Additional examples of softwaredistribution platforms include third party software distributionplatforms, such as the Baidu App Store, GetJar, and Handango. Anotherway the nefarious individual can attempt to propagate malware is byposting it on a message/forum site, such as site 136. In this scenario,the nefarious individual again hopes that unsuspecting users of site 136will download and install the malicious application/content 130. Yetanother way for the nefarious individual to attempt to propagate malware130 is to attach it to an email message and hope that the recipient(e.g., the owner of device 104) will open the attachment and, forexample install the program. Yet another way for the nefariousindividual to attempt to propagate malware 130 is to include it in anadvertising company's ad network (e.g., mobile ad network 132) and hopethat the user will install the promoted program.

In an example shown in FIG. 1, client devices 104 and 106 are asmartphone and a tablet (respectively) present in an enterprise network110. Client device 108 is outside enterprise network 110. In thisexample, client device 104 runs an Android-based operating system (OS),client device 106 runs Windows 10, client device 108 runs a version ofthe Microsoft Windows® OS, and client device 108 runs a version of iOS.Each of the devices shown can be protected using techniques describedherein. Other devices, running other operating systems, whethermobile-oriented or not, can also be protected using the techniquesdescribed herein. As will be described in more detail below, in variousembodiments, endpoint protection is provided to each of devices 104,106, 107, and 108. Such endpoint protection can be provided in a varietyof manners and by a variety of entities, whether working individually,or in cooperation. For example, each of devices 104, 106, 107, and 108can execute a program, or set of programs, providing endpoint protectionservices. The endpoint protection can be natively incorporated into theoperating system running on the device (e.g., by the author of theoperating system, the device manufacturer, or another appropriate entityincluding the operator of service 122), and can also be appliedaftermarket (e.g., with the user or another appropriate entityinstalling an endpoint protection application or application suite onthe device). Some examples of functionality provided by endpointprotection include local (to the device) firewalling, maliciousapplication scanning, data loss/leakage prevention, sandboxing, etc. Theendpoint protection application can make use of a variety of resources,including white/blacklists of MD5 or other hashes of known good/badapplications, stored in an appropriate storage location accessible tothe endpoint protection application. The white/blacklists can beprovided by a third party, such as an operator of service 122, and canalso be configured by an administrator, by the user, etc., asapplicable. In accordance with techniques described herein, and asdescribed in more detail below, endpoint protection can integrate withremote services (e.g., offered by cloud security service 122) inproviding protection to the device on which the endpoint protection isexecuting. In particular, endpoint protection applications 154, 156,158, and 160 can monitor their respective local devices for variousbehaviors (e.g., using the BTP engine as will be further describedbelow), and can determine whether such behaviors aresuspicious/malicious based at least in part on information obtained fromservice 122 (e.g., threat logic based on a set of rules received fromservice 122 or configured by an IT/security admin for enterprise 110 aswill be further described below). A variety of actions can be taken inresponse to such detections, as also described in more detail below.Using techniques described herein, the cloud security service 122provides review services for applications, content, and/or other formsof malware, and the endpoint protection applications restrict thebehaviors of the application/content (when executing in respectivedevices) to only those that have been reviewed and confirmednon-malicious by the cloud security service. For example, if anapplication, such as malicious application 130, attempts to bypass theanalysis performed by the cloud security service, the malicious natureof the application will be caught by the endpoint protection applicationwhen the malicious application attempts to engage in malicious behavioron end user devices. Additional benefits include the endpoint protectionapplication collecting additional behavior information from executingapplications and reporting behaviors back to the cloud security service,and helping system administrators better determine which kinds ofbehaviors a given application will engage in after installation.

In some embodiments, data appliance 102 is configured to enforcepolicies regarding communications between clients such as clients 104,106, 107, and 108, and nodes outside of enterprise network 110 (e.g.,reachable via external network 118). Examples of such policies includeones governing traffic shaping, quality of service, and routing oftraffic. Other examples of policies include security policies such asones requiring the scanning for threats in incoming (and/or outgoing)email attachments, website downloads, files exchanged through instantmessaging programs, and/or other file transfers. In some embodiments,appliance 102 is also configured to enforce policies with respect totraffic that stays within enterprise network 110. In some embodiments,other devices are included in network 110, such as a mobile devicemanagement (MDM) server 146, which is in communication with dataappliance 102.

As shown, MDM server 146 communicates with devices (e.g., 104 and 106)to determine device status and to report (e.g., periodically) suchdevice status information to data appliance 102. MDM server 146 can beconfigured to report the presence of known malicious applicationsinstalled on devices such as device 104/106, and/or can be configured toreceive indications of which applications are malicious (e.g., fromappliance 102, from service 122, or combinations thereof). In someembodiments, data appliance 102 is configured to enforce polices againstdevices 104 and 106 based on information received from MDM server 146.For example, if device 106 is determined to have malware installed on it(e.g., by MDM server 146), data appliance 102 (working in cooperationwith MDM server 146) can deny client 106 access to certain enterpriseresources (e.g., an Intranet) while allowing device 104 (which does nothave malware installed upon it) access to those resources. In variousembodiments, data appliance 102 is configured to enforce policiesagainst devices 104 and 106 based on information received from thedevices themselves (e.g., as provided by endpoint security applications154 and 156 respectively running on devices 154 and 156 to appliance102). While MDM server 146 is shown in the environment of FIG. 1, thepresence of an MDM server 146 is not required to implement variousembodiments of techniques described herein, whether inside or outside anenterprise network setting. As one example, client device 107 (and isexecuting end point protection application 160) and client device 108(which is outside enterprise network 110, and is executing end pointprotection application 158) can benefit from techniques describedherein, as will be described in more detail below.

FIG. 2 illustrates a data appliance in accordance with some embodiments.The example shown is a representation of physical components that areincluded in appliance 102, in various embodiments. Specifically,appliance 102 includes a high performance multi-core CPU 202 and RAM204. Appliance 102 also includes a storage 210 (such as one or more harddisks), which is used to store policy and other configurationinformation, as well as URL information. Data appliance 102 can alsoinclude one or more optional hardware accelerators. For example, dataappliance 102 can include a cryptographic engine 206 configured toperform encryption and decryption operations, and one or more FPGAs 208configured to perform matching, act as network processors, and/orperform other tasks.

Appliance 102 can take a variety of forms. For example, appliance 102can be a single, dedicated device (e.g., as shown), and can also be aset of devices. The functionality provided by appliance 102 can also beintegrated into or executed as software on a general purpose computer, acomputer server, a gateway, and/or a network/routing device. Forexample, in some embodiments, services provided by data appliance 102are instead (or in addition) provided to client 104 (or clients 106,107, or 108) by an agent or other software executing at least partiallyon client 104 (or clients 106, 107, or 108), such as endpoint protectionexecuting on any/all of clients 104, 106, 107, and 108 (e.g.,applications 154, 156, 158, and 160, respectively). In an exampleimplementation, the endpoint agents (e.g., such as a commerciallyavailable security endpoint agent, such as the Palo Alto Networks, Inc.provided Traps® agent) protects endpoints by preventing known andunknown malware from running on those endpoints and by halting anyattempts to leverage software exploits and vulnerabilities (e.g., usingthe disclosed BTP engine and associated new techniques as furtherdescribed below). The endpoint agents can enforce a security policy forthe enterprise (e.g., enterprise 110). When a security event occurs onan endpoint, the agent collects forensic information about that eventthat you can use to analyze the incident.

Whenever appliance 102 is described as performing a task, a singlecomponent, a subset of components, or all components of appliance 102may cooperate to perform the task. Similarly, whenever a component ofappliance 102 is described as performing a task, a subcomponent mayperform the task and/or the component may perform the task inconjunction with other components. In various embodiments, portions ofappliance 102 are provided by one or more third parties. Depending onfactors such as the amount of computing resources available to appliance102, various logical components and/or features of appliance 102 may beomitted and the techniques described herein adapted accordingly.Similarly, additional logical components/features can be added to system102 as applicable.

Suppose data appliance 102 intercepts an email sent by system 120 todevice 104 to which a copy of malware 130 has been attached. (As analternate, but similar scenario, data appliance 102 could intercept anattempted download by device 104 of malware 130 from platform 134 orsite 136.) Data appliance 102 determines whether a signature for theattachment (i.e., malware 130) is present on data appliance 102. Asignature, if present, can indicate that the attachment is known to besafe, and can also indicate that the attachment is known to bemalicious. If no signature for the attachment is found, in someembodiments, data appliance 102 is configured to provide the attachment(malware 130) to a malware analysis module 112 for real-time analysis.As an alternate example, endpoint protection (e.g., any of 154, 156,158, and 160) can detect when an attempt is made to install software(e.g., malware 130) on a device, and communicate (whether directly, orindirectly—such as through data appliance 102) with service 122 todetermine information about the application, in accordance withtechniques described herein.

As will be described in more detail below, a combination of static anddynamic analysis can be performed on applications such as application130 to determine whether they are malicious. The analysis of malware 130can be performed by a malware analysis module 112 included in dataappliance 102, implemented as a set of one or more programs executing onappliance 112. Instead of or in addition to on-premise analysis,appliance 102 can also send a copy of malware 130 to cloud securityservice 122 for analysis. Cloud security service 122 can also (orinstead) obtain copies of samples (e.g., mobile applications and/orother types of applications, web content/pages, shared files of variousfile types/formats, as well as other potential source of malware, etc.)for evaluation from sources other than data appliance 102. As oneexample, cloud security service 122 can include a crawler 138 configuredto periodically crawl platform 134 and/or site 136, looking for new orupdated applications, web pages, etc. Such malware samples (an exampleof which is malware 130) can then be analyzed by cloud security service122. In some embodiments, platform 134 and/or site 136 make copies ofmalware samples available to cloud security service 122 via anApplication Programming Interface (API) made available by service 122,instead of or in addition to crawler 138 obtaining such copies. Further,as will be described in more detail below, in various embodiments,devices 104, 106, 107, and/or 108 make copies of malware samples such asmalware 130 available to cloud service 122 (or as applicable, to malwareanalysis module 112), such as in conjunction with a local user attemptto install a copy of the malware sample on the receptive device.

Copies of received malware samples (i.e., awaiting analysis) are storedin storage 142 and analysis is commenced (or scheduled, as applicable).As will be described in more detail below, results of the analysis (andadditional information pertaining to the malware samples) are stored indatabase 140. In some embodiments, cloud security service 122 comprisesone or more dedicated commercially available hardware servers (e.g.,having multi-core processor(s), 8G+ of RAM, gigabit network interfaceadaptor(s), and hard drive(s)) running typical server-class operatingsystems (e.g., Linux). In various embodiments, service 122 isimplemented across a scalable infrastructure comprising multiple suchservers, solid state drives, and/or other applicable high-performancehardware. Cloud security service 122 can comprise several distributedcomponents, including components provided by one or more third parties.For example, portions or all of cloud security service 122 can beimplemented using the Amazon Elastic Compute Cloud (EC2) and/or AmazonSimple Storage Service (S3). Further, as with data appliance 102, whencloud security service 122 is referred to as performing a task, such asstoring data or processing data, it is to be understood that asub-component or multiple sub-components of cloud security service 122(whether individually or in cooperation with third party components) maycooperate to perform that task. As one example, cloud security service122 can optionally perform its analysis in cooperation with one or morevirtual machine (VM) servers, such as VM server 124.

An example of a virtual machine server is a physical machine comprisingcommercially available server-class hardware (e.g., a multi-coreprocessor, 4+ Gigabytes of RAM, and one or more Gigabit networkinterface adapters) that runs commercially available virtualizationsoftware, such as VMware ESXi, Citrix XenServer, or Microsoft Hyper-V.In some embodiments, the virtual machine server is omitted. Further, avirtual machine server may be under the control of the same entity thatadministers cloud security service 122, but may also be provided by athird party. As one example, the virtual machine server can rely on EC2,with the remainder portions of cloud security service 122 provided bydedicated hardware owned by and under the control of the operator ofcloud security service 122. As will be explained in more detail below,virtual machine server 124 is configured to provide one or more virtualmachines 126-128 for emulating various types of endpoint devices. Thevirtual machines can execute a variety of operating systems and/orversions thereof. Observed behaviors resulting from executing samples(e.g., applications, content, etc.) in the virtual machines are loggedand analyzed (e.g., for indications that the sample is malicious). Insome embodiments, the log analysis is performed by the VM server (e.g.,VM server 124). In other embodiments, the analysis is performed at leastin part by other components of service 122, such as coordinator 144.

In some embodiments, cloud security service 122 makes available theresults of its analysis of potential malware/malware activity via a listof signatures (and/or other identifiers) to appliance 102, to any/all ofdevices 104, 106, 107, and 108 (and/or to MDM server 146) as part of asubscription. For example, service 122 can send a content package thatidentifies malware periodically (e.g., daily, hourly, or some otherinterval, and/or based on an event based on a policy). An examplecontent package includes a listing of identified malware (e.g.,information in the content package for malware apps can include an apppackage name, an app hash code for uniquely identifying the app, and amalware name for each identified malware app). The subscription cancover the analysis of just those files intercepted by data appliance 102and sent to cloud security service 122 by data appliance 102, and canalso cover signatures of all malware known to cloud security service122. Cloud security service 122 is configured, in various embodiments,to provide security services to entities in addition to or instead of anoperator of data appliance 102. For example, cloud security service 122can similarly perform the disclosed techniques for detecting maliciousactivity on an endpoint based on real-time system events, which will befurther described below. As another example, the dynamic analysis ofsamples performed using cloud security service 122 can be utilized togenerate new threat logic (e.g., a set of threat rules specified in aprogramming/scripting language, such as JavaScript or anotherprogramming/scripting language can be utilized, such as Python or Lua)that can be efficiently deployed as content updates to endpoint agents104, 106, 107, and 108 as will also be further described below.

In the event malware 130 is determined to be malicious (whether by cloudsecurity service 122 or by data appliance 102), appliance 102 can beconfigured to automatically block the file download based on theanalysis result. As will be described in more detail below, in variousembodiments, endpoint protection executing on devices 104, 106, 107, and108 can similarly block a user of the respective devices from, forexample, completing an install of the malicious application (insteadof/in addition to relying on appliance 102 to prevent the downloading ofthe malicious application) and/or downloading the malicious file ormalicious other content. Further, a signature can be generated formalware 130 and distributed (e.g., to other data appliances) toautomatically block future file transfer requests to download thecontent/file determined to be malicious.

In some cases, sample is malicious, but may not be (at least initially)flagged as malicious by cloud security service 122 (or malware analysismodule 112 or other appropriate entity as applicable) during analysis.One reason for this is that malware authors increasingly usesophisticated techniques to conceal the malicious behaviors of theirmalware in order to evade detection by services such as service 122. Asan example, a malicious application may attempt to reserve its maliciousbehavior for when the malicious application is executing on a victim'sdevice. Even where service 122 can (e.g., through static and dynamicanalysis) identify some of the concealment techniques on the part ofmalware authors, new approaches by malware authors to evading detectionof the malicious nature of their applications are continuously beingdeveloped.

Suppose a typical malware analysis system can allocate five minutes toevaluating an arbitrary application for maliciousness (e.g., due toavailable resources, amounts of delay tolerated by users awaitingmaliciousness verdicts, etc.). A malware author might suppress theexpression of malicious aspects of an application for an hour, or a day,or longer, from the point the application is first installed and run onan end user device, and evade detection of the application'smaliciousness accordingly. The amount of time the malicious behavior issuppressed can be programmed into the application (e.g., as a “sleep( )”or other routine), and can also rely on an external instruction, such asby having the application contact a command and control server. Asanother example of an evasion technique, malware can be configured todetect whether it is running in a virtualized environment (e.g., underthe assumption that the malware is being observed for security analysisreasons in a VM such as VM 126) and only take benign actions if so. Asyet another example, a malicious application may initially install witha minimum set of functionality, and then, once installed, downloadadditional, malicious elements (e.g., as an update, as a patch, as anadditional library, etc.). If an analysis system only analyzes theapplication in its initial form, malicious behaviors (e.g., asaccomplishable by the malware once it has updated/patched itself) may goundetected. As will be described in more detail below, using techniquesdescribed herein, devices such as devices 104, 106, 107, and 108 can beprotected from malicious applications, including those not initiallydetermined to be malicious (e.g., those not flagged as malicious by aninitial analysis by service 122) by performing the disclosed techniquesfor detecting malicious activity on an endpoint based on real-timesystem events, which will be further described below.

Analyzing Potential Malware

FIG. 3 illustrates an example of logical components included in a systemfor performing static and dynamic analysis on samples (e.g., potentialmalware, which may be files, applications, web pages/content, and/orvarious other content) in accordance with some embodiments. In variousembodiments, system 300 is implemented using a single device. Forexample, the functionality of system 300 can be implemented on dataappliance 102 which includes an analysis module 112. System 300 can alsobe implemented, collectively, across multiple distinct devices. Forexample, the functionality of system 300 can be provided by cloudsecurity service 122.

As will be described in more detail below, in various embodiments,system 300 is configured to perform a hybrid, two part analysis onsamples. First, static analysis is performed, in part to check, forexample, the capabilities of the sample (e.g., for an application, suchcapabilities can be potential avenues for being malicious). Then,dynamic analysis is performed to check whether the sample performs anyundesired and/or malicious activities/behaviors during execution (e.g.,if the application actually uses the capabilities maliciously). Thehybrid approach helps improve the accuracy of malware detection, whilelowering the false positive rate of mislabeling benign samples asmalware (e.g., due to harmless but poor programming techniques on thepart of the application's author). An initial verdict pertinent to thesample can be made based on both the samples content (e.g., where theapplication, file, web page, or other content includes a URL verified tobe a malicious website), and on the context in which it behaves (e.g.,whether the usage of a suspicious capability by the application is madeaware to an end user or is performed silently in the background). Aswill be described in more detail below, in various embodiments, in theevent that the verdict is that the sample being evaluated is malicious,the sample can be blocked from being accessed by and/or installed on adevice such as any of devices 104, 106, 107, and 108 (e.g., by endpointprotection executing on the implicated devices). In the event that theverdict is not malicious, behaviors observed during analysis can berecorded and used to enforce limits on the sample (e.g., application)once installed, reducing the ability of a malware to engage in behaviorsnot observed during analysis.

In various embodiments, system 300 makes use of lists, databases, orother collections of known safe content and/or known bad content(collectively shown in FIG. 3 as collection 314). Collection 314 can beobtained in a variety of ways, including via a subscription service(e.g., provided by a third party) and/or as a result of other processing(e.g., performed by data appliance 102 and/or service 122). Examples ofinformation included in collection 314 are: URLs of known maliciouswebsites; URLs of known safe websites; signatures, hashes, and/or otheridentifiers of known malicious applications; and signatures, hashes,and/or other identifiers of known safe applications; and signatures,hashes, and/or other identifiers of known malicious files (e.g. Androidexploits files).

Ingestion

In some embodiments, when a new sample (or, as applicable, an updatedone) is received for analysis (e.g., an existing signature associatedwith the sample is not present in system 300), it is added to processingqueue 302. In the following example, suppose the application is called“game.apk,” (the malicious game 130) but that it is not yet knownwhether or not the game is malicious. As explained above, a sample canbe received for analysis in a variety of ways. As one example, a samplecan be received by data appliance 102 for analysis when data appliance102 intercepts an email or other data transmission intended for device104 that includes the application. One additional example of the receiptof a sample is by service 122 of the application from data appliance102, platform 134, or site 136 for analysis (e.g., via an API). Anotheradditional example of receipt of a sample is crawling by service 122 ofsystems such as platform 134 or site 136. Yet another way a sample canbe received for analysis is at the direction of endpoint protectionexecuting on a device onto which an attempt to install the applicationis being made. For example, when users of any of devices 104, 106, 107,and 108 attempt to install new applications on their respective devices,endpoint protection executing on the devices can, upon notification ofthe attempted install, send a copy of the application to service 122(e.g., via an API), can send a hash of the application to service 122(e.g., to determine whether information pertinent to the application isalready present e.g., in database 140), can provide a link to the sourceof the application (e.g., where the user is attempting to install theapplication from platform 134 or site 136), etc.

Static Analysis

Coordinator 304 monitors the queue, and as resources (e.g., a staticanalysis worker) become available, coordinator 304 fetches anapplication from queue 302 for processing (e.g., fetches game.apk). Inparticular, coordinator 304 first provides the application to staticanalysis engine 306 for static analysis. In some embodiments, one ormore static analysis engines are included within system 300, wheresystem 300 is a single device. In other embodiments, static analysis isperformed by a separate static analysis server that includes a pluralityof workers (i.e., a plurality of instances of static analysis engine306).

One example of how static analysis can be performed, using game.apk (anAndroid application) as an example is as follows. Similar approaches canbe used for applications executable on other platforms. First, thestatic analysis engine 306 uses a tool, such as Android apktool, toreverse game.apk into an intermediate source code form. The output ofthe reversing operation is, in some embodiments, a set of .smalifiles—the direct output of the disassembly from Java virtual machinelanguage, and other resources files included in the game.apk file.

The static analysis engine obtains general information about theapplication, and includes it (along with heuristic information describedbelow) in a static analysis report 308. The report can be created by thestatic analysis engine, or by coordinator 304 (or by another appropriatecomponent) which can be configured to receive the information fromstatic analysis engine 306. In some embodiments, the collectedinformation is stored in a database record for the application (e.g., indatabase 140), instead of or in addition to a separate report 308 beingcreated (i.e., portions of the database record form the report 308).Examples of collected information include: the package name, shared UID,APK file signer information, permissions claimed, and sensitive APIcalls included in the source (e.g., sending or erasing SMS messages,accessing the phonebook, and tracking user location changes). The staticanalysis engine also collects and stores information pertaining to therunning context of the application, such as: the minimum version of theAndroid OS required to run the application (the minimum SDK version),and the sensors it will have access to.

The static analysis engine also retrieves (e.g., from database 140) aset of heuristic rules to be applied on the .smali code and the resourcefiles. In particular, static analysis engine 306 determines which rulesare triggered (also referred to as “features hit”) by the source code.Examples of features include the following (where an example of “thereceived APK” is “game.apk”): contains known malicious APK file, filetype mismatch, contains malicious executable files, requires abnormalpermissions, contains malicious URL(s), etc.

The static analysis engine stores the results of the rule testing adatabase (e.g., database 140) in the record associated with theapplication being tested (and/or includes the results in report 308 asapplicable). In some embodiments, the static analysis engine also formsa verdict with respect to the application (e.g., “safe,” “suspicious,”or “malicious”). As one example, the verdict can be “malicious” if evenone “malicious” static feature is present in the application. As anotherexample, points can be assigned to each of the features (e.g., based onseverity if found; based on how reliable the feature is for predictingmalice; etc.) and a verdict can be assigned by static analysis engine306 (or the coordinator, if applicable) based on the number of pointsassociated with the static analysis results. A “safe” verdict can bealternately considered as a “nothing malicious (or suspicious, asapplicable) detected” verdict. The potential exists, for example, that aclever malware author has crafted the app in such a way as to appearbenign even when it is not. As will be described in more detail below,characteristics of an app deemed “safe” by system 300 can be recordedand used to generate rules that govern how the app, once installed on anend user device, is permitted to operate.

In some cases, an application may appear “suspicious” to static analysisengine 306 due to poor programming choices made by a harmlessprogrammer, rather than a malicious one. As one example, the programmermay have named an executable that handles playing of an MP3 file with a“.mp3” extension. This sort of file type mismatch (i.e., that anexecutable is incorrectly labeled with a non-executable extension) couldindicate malicious behavior (i.e., a malicious individual is trying tohide a malicious executable through misnaming the filename). Here,however, the file was inadvertently mislabeled. Static analysis engine306 notes (e.g., with rule “File Type Mismatch” being included in thestatic analysis report) that there is a “suspicious” aspect to the filewhich warrants additional investigation during dynamic analysis to reacha conclusion as to whether the application is benign or malicious.

In some embodiments, static analysis engine 306 will conclude that theapplication will crash (and/or cause the virtual machine to crash) ifexecuted. As one example, static analysis engine 306 can performintegrity checking and determine that a file is missing, corrupted,unsigned, etc. In this scenario, dynamic analysis can be skipped (e.g.,with static analysis noting in report 308 that the application willcrash if an attempt is made to install/execute it).

Dynamic Analysis

Once the static analysis is complete, coordinator 304 locates anavailable dynamic analysis engine 310 to perform dynamic analysis on theapplication. As with static analysis engine 306, system 300 can includeone or more static analysis engines directly. In other embodiments,dynamic analysis is performed by a separate dynamic analysis server thatincludes a plurality of workers (i.e., a plurality of instances ofdynamic analysis engine 310).

Each dynamic analysis worker manages a device emulator (e.g., running ina virtual machine, to emulate a device platform, such as a personalcomputing platform, such as Microsoft Windows OS®, Linux OS, Apple MacOS®, etc., and/or a mobile computing platform, such as Apple iOS®,Google Android®, etc.). Results of the static analysis (e.g., performedby static analysis engine 306), whether in report form (308) and/or asstored in database 140, or otherwise stored are provided as input todynamic analysis engine 310. The static report information is used tohelp customize the type of dynamic analysis performed by dynamicanalysis engine 310, conserving resources and/or shortening the timerequired to evaluate an application. As one example, if static analysishas concluded that the application does not have the ability to accessSMS messages, during dynamic analysis, the receipt of SMS messages willnot be simulated in some embodiments. As another example, if staticanalysis has concluded that the application has the ability to accessGPS information, during dynamic analysis, various changes in location ofthe device will be simulated. However, if the application lacks theability to access GPS information, in some embodiments no locationchanges will be simulated (reducing the amount of time/computingresources needed to complete dynamic analysis). As yet another example,dynamic analysis engine 310 will determine which emulator(s) to runbased on the minimum operating system version number required by theapplication (and determined during static analysis). If the minimumversion number is Android 4.0, dynamic analysis engine 310 will launchan Android emulator having that version number (and, in someembodiments, will not attempt to emulate a lower version of Android). Ifthe minimum version number is Android 2.3, multiple emulators can beused to evaluate the application (e.g., Android 2.3, and any higherversioned emulators, such as Android 4.0). Where multiple emulators areused, a single dynamic analysis engine can manage all of the emulators(whether in sequence or in parallel), or multiple dynamic analysisengines can be used (e.g., with each managing its own emulator), asapplicable.

The dynamic analysis engine/worker begins analysis by preparing andsetting up the running environment for the application to be tested.Examples of operations carried out by the dynamic analysis engine/workerat this point include: (1) determining which system services should bestarted (e.g., simulated motion sensor readings and simulated locationchanges); and (2) determining what set of simulated user operationsshould take place (e.g., performed after installation, in sequence).

The dynamic analysis engine/worker loads an appropriate emulator (e.g.,Android version 2.3) and installs the application to be analyzed. Theemulators used by malware analysis system 300 are instrumented. Forexample, they are configured to log activities as they occur in theemulator (e.g., using a customized kernel that supports hooking andlogcat). Further, network traffic associated with the emulator iscaptured (e.g., using pcap).

The application is executed and various applicable actions (e.g.,selected based on static analysis report 308) are performed (e.g., bythe dynamic analyzer executing commands via an Android Debug Bridge(“adb”) connection and/or through the use of a service coordinatorincluded in the modified emulator and configured to orchestrate thesimulation of user events such as button presses as commanded by thedynamic analysis engine). As one example, if the application wasdetermined during static analysis to have access to locationinformation, changes in location will be simulated in the emulator andany resulting behaviors logged. In some embodiments the log data isstored as a temporary file on system 300.

In some embodiments, dynamic analysis is performed in two stages. Inparticular, after the application has been installed and executed (withassociated simulated information/events) and a first log file is created(e.g., “logcat1.txt”), a reboot of the emulator is performed and theapplication is launched and interacted with again, resulting in a secondlog file (e.g., “logcat2.txt”). Dynamic analysis engine 310 evaluatesboth log files, along with any network traffic captured during the twostages (e.g., using pcap).

Examples of features that can be detected during dynamic analysisinclude the following (where an example of “the received APK” is again“game.apk”): connecting to unknown and/or malicious websites, create amalicious file, load a malicious file, access hidden file or hiddenfolder, change file permissions, and/or various other activities thatcan be monitored and detected during execution, including the disclosedmonitored activities that can be detected using the BTP engine, such asfurther described below.

As with the static analysis engine, the dynamic analysis engine storesthe results of the rule testing in the database in the record associatedwith the sample (e.g., application, content, etc.) being tested (and/orincludes the results in report 312 as applicable). In some embodiments,the dynamic analysis engine also forms a verdict with respect to thesample (e.g., safe, suspicious, or malicious). As one example, theverdict can be “malicious” if even one “malicious” dynamic feature ispresent in the sample. As another example, points can be assigned toeach of the features (e.g., based on severity if found; based on howreliable the feature is for predicting malice; etc.) and a verdict canbe assigned by dynamic analysis engine 306 (or the coordinator, ifapplicable) based on the number of points associated with the staticanalysis results. As with static analysis, a “safe” verdict (e.g.,determined during dynamic analysis) can be alternately considered as a“nothing malicious (or suspicious, as applicable) detected” verdict. Thepotential exists, for example, that a clever malware author has craftedthe malware in such a way as to appear benign even when it is not. Aswill be described in more detail below, characteristics of an malwaredeemed “safe” by system 300 can be recorded and used to generate rulesthat govern how the malware, once installed/opened on an end userdevice, is permitted to operate.

In some embodiments, a final verdict associated with the sample isassessed (e.g., based on a combination of report 308 and report 312) bycoordinator 304.

Embodiments of a Behavior Threat Protection Engine Architecture forDetecting Malicious Activity on an Endpoint Based on Real-Time SystemEvents

FIG. 4 illustrates an architecture of a Behavior Threat Protection (BTP)engine for detecting malicious activity on an endpoint based onreal-time events in accordance with some embodiments. In one embodiment,BTP engine shown in FIG. 4 is a component of endpoint security agents154, 156, 158, and 160 (e.g., and/or can be implemented using aninstrumented emulation environment performing dynamic analysis, such assimilarly described above with respect to cloud security service 122 ofFIG. 1).

For example, the endpoint agent can perform behavioral threat protectionby continuously monitoring various activities on the endpoint toidentify and analyze a set of events (e.g., real-time events), such aschains of events (e.g., referred to herein as causality event chains),as opposed to just a single event. This enables the endpoint agent todetect malicious activity in the set of events (e.g., a chain/pattern ofevents) that may otherwise appear legitimate if each of the events wasmerely inspected individually.

Referring to FIG. 4, the BTP engine (e.g., a component/module ofendpoint security agents, such as the endpoint security agents describedabove) includes an Infrastructure module 402 that includes an OS EventsAggregator 404 for various events that are monitored and aggregated. AFile System (FS) event and filter module 406 monitors and filtersvarious file system events (e.g., file create, file delete, file open,file read, file write, and/or various other file system related eventsetc.) on the endpoint. A Process IDs (PIDs) event and filter module 408monitors and filters various process related events (e.g., processcreate, process kill, and/or various other process related events) onthe endpoint. A Network (Net) event and filter module 410 monitors andfilters various process related events (e.g., network connection open,network connection close, and/or various other network related events)on the endpoint. An OS private Application Programming Interfaces (APIs)event and filter module 410 monitors and filters various OS private APIrelated events (e.g., system OS kernel API calls, and/or various otherOS private API related events) on the endpoint.

OS Events Subscription API 414 receives each of these aggregated andfiltered events, which can be normalized (e.g., into a canonicalizedformat and flow) and then provided to an Event Listener 420 of an EventsObserver (observd) module 418. In an example implementation, each ofthese filters can be configured using, for example, a YAML implementedfiltering policy, which is received via a Security Connector 424 andFilter Config API 426. Various filtering policies can be configured tofocus on events that may be more relevant for detecting security relatedevents (e.g., to filter out/disregard events that may be too noisyand/or may not be useful indicators for malware detection usingcausality event chains). An example FS filter can include aconfiguration for filtering out/disregarding file read operations (e.g.,except for in some cases, reading of specified files that may be moresensitive files or as another example a threshold number of reads to adatabase, etc.). For example, a number of events monitored in a typicalendpoint's kernel and user space can be very high. As such, in order tofacilitate a more efficient event matching process, the BTP engineincludes various configurable filters for filtering out some of theevents (e.g., uninteresting registry key hives, duplicate file-reads,etc.). In an example implementation, the disclosed event-based filteringtechniques implement a Last Recently Used (LRU) caching algorithm (anddata structure). Filtering values can be configurable by content (e.g.,in an Agent configuration package) and can be changed by securityanalysts (e.g., a set of malware researchers, such as in a researchgroup for a commercial security service provider) (e.g., and othersources) on demand.

Event Listener 424 provides the normalized set of filtered events to anEvent Scheduler 422 that schedules event-based security detectionanalysis to be performed on the normalized set of filtered events usinga Security Connector 424. Security Connector 424 applies a securityrules (e.g., behavioral threat rules) to the normalized set of filteredevents to determine if there is a match with one or more of the securityrules. Specifically, Security Connector 424 performs a pattern matchanalysis using a Lookup Tree 432 (e.g., implemented as a Rete tree thatis compiled to provide optimized detection logic based on optimizeddecision trees) of a Dynamic Security Engine module 430. If there is amatch, then an Action & Report 428 can be performed based on theconfigured action and/or reporting in the configured security rule thatwas determined to match based on the set of monitored real-time eventson the endpoint.

In an example implementation, these security rules (e.g., behavioralthreat rules) can be written in a scripting/rules-based language, suchas JavaScript and stored as a JSON file as further described below, andcan be matched based on regular expression (REGEX) matching using theCLIPs engine as will be further described below. The security rules canbe created by a security analyst at a security service provider and/orby an enterprise IT/network/security admin and stored in a content datastore (e.g., database) 434. The security rules can be provided to aCompiler 436 via an API of Dynamic Security Engine 436 for compilingRete tree 432 as shown in FIG. 4.

In an example implementation, the causality chains that are maliciouscan be detected using the disclosed new behavioral threat rules that canbe compiled and stored in a Rete tree for efficient pattern matching ofreal-time events on the endpoint. Any new and/or modified security rules(e.g., behavioral threat rules) can be efficiently provided as contentupdates (e.g., that can be received and compiled duringexecution/runtime, without necessarily requiring binary/code updates tothe endpoint agents). Also, enterprise IT/network/security admins canconfigure the action(s) and/or reporting responses performed by theendpoint agent when a security rule pattern match is detected to performthe desired responsive action (e.g., kill the process, close the networkconnection, report the actions to the user and/or to an enterpriseIT/network/security admin, quarantine the Causality Group Owner (CGO)which initiated the activity when the match was detected, etc.).

In this example, when the endpoint security agent (e.g., BTP engine)detects a chain of real-time monitored events that match a behaviorthreat rule, then the endpoint security agent performs the configuredaction and reports details about the activities that led to the securityevent (e.g., to alert on the attack, block the attack, report on/log theattack, notify an endpoint user of the attack via, for example, a pop-upwindow notification on the endpoint user interface, quarantine theendpoint being attacked, etc.). In an example implementation, users(e.g., IT/network/security admins) can review the entire causality chainup to the Causality Group Owner (CGO) on an Analysis tab of a securityevent or console of the endpoint security solution. In some cases, afteranalyzing the flow of events, the IT/network/security admin believesthat the behavior is legitimate, then they can configure a policyexception (e.g., Exception rule such as further described below) fromthe matched pattern of events to disable the behavior rule on theendpoint.

As such, the disclosed techniques facilitate a more flexible frameworkfor detecting malware based on causality chains that facilitates updatesusing new detection logic (e.g., security rules including behavioralthreat rules that can be efficiently distributed/pushed to the endpointsecurity agents as content updates without requiring an endpointsecurity agent upgrade or any downtime to the enterprise customers).Specifically, the new behavioral threat rules can be scripted (e.g., inJavaScript or another scripting language) to specify the causalitychains for detecting malware activities on an endpoint based on aparticular chain of real-time events (e.g., various events can bemonitored on the endpoint device/system, such as file system operations,process creation and other process related events, injection operations,registry related operations, Remote Procedure Call (RPC) relatedoperations, network connection related operations, and/or various othertypes of events), such as will be further described below.

Accordingly, in this example implementation, the disclosed techniquesprovides a more flexible framework for detecting malware thatfacilitates an improved mechanism for deploying new detection logicrapidly and without necessarily requiring binary/code updates. Inaddition, the new detection logic (e.g., new detection rules includingdynamic event-based analysis rules, such as behavioral threat rules) canbe scripted to facilitate content updates for detecting new forms ofmalware (e.g., the rules can be written, in the form of a script, andthe scripts can be remotely updatable to allow for rapid updatingwithout having to necessarily perform binary/code updates to endpointsecurity agents). Moreover, the new detection logic can be scripted bysecurity vendors (e.g., security analysis) as well as customersthemselves (e.g., IT/network/security admins). Thus, the disclosed BTPdynamic engine incorporates both static and runtimecharacteristics/dynamic behavioral analysis of malware based onreal-time events monitored on an endpoint device and can be remotelyupdated via a scriptable language, which allows for much greaterflexibility in efficiently distributing new detection logic via contentupdates (e.g., incremental content updates that describe rules,features, and alerting events for matches for detecting malware, whichcan be provided as a relatively small JSON file that can bedistributed/downloaded to the endpoint security agents) and reduces theresponse time to new threats as they come for the deployed endpointsecurity agents. Moreover, the disclosed BTP engine also allows otherentities (e.g., IT/network/security admins of enterprise customers) toutilize this engine to push content of their own, thereby dramaticallyincreasing the number of people who can contribute detection logic tothe product.

As also shown in FIG. 4, for testing the BTP engine with existing and/ornew/modified security rules (e.g., behavioral threat rules), a testingframework can (optionally) be provided. The testing framework caninclude an events injector 416 to inject a set of events for testing theBTP engine as well as an API Tester for testing the pattern matching ofthe injected events with the existing and/or new/modified securityrules.

FIG. 5 illustrates an architecture flow of a Behavior Threat Protection(BTP) engine for detecting malicious activity on an endpoint based onreal-time events in accordance with some embodiments. In one embodiment,the BTP engine, such as shown in FIG. 4, and the architecture flowperformed using the BTP engine as shown in FIG. 5 can be implemented asa component of endpoint security agents 154, 156, 158, and 160 (e.g.,and/or can be implemented using an instrumented emulation environmentperforming dynamic analysis, such as similarly described above withrespect to cloud security service 122 of FIG. 1). For example, theendpoint security agent can perform behavioral threat protection bycontinuously monitoring various activities (e.g., real-time events) onthe endpoint to identify and analyze chains of events (e.g., alsoreferred to herein as causality event chains or simply causality chains)as opposed to attempting to make security determinations based on only asingle event. As similarly described above, this enables the endpointsecurity agent to detect malicious activity based on a monitored chainof events (e.g., a pattern of events) that may otherwise appearlegitimate if each of the events were inspected individually/inisolation.

In one embodiment, the BTP engine is implemented in a high-levelprogramming language as a cross-platform endpoint security component,such as for the Microsoft Windows® operating systems, Apple MacOS®operating systems, Linux operating systems, etc. In an exampleimplementation, the BTP engine is implemented using an open source rulesbased tool, such as the C Language Integrated Production System (CLIPS)rule based expert system technology tool (e.g., CLIPS is an open source,public domain software tool that deals with rules and facts and isavailable at, for example,https://sourceforge.net/p/clipsrules/discussion/776945/thread/e001210c/?limit=25)as will be further described below.

In another embodiment, the BTP engine can be extended by externallibraries to add capabilities (e.g., such as an optimized regexpmatching library). In this example implementation, the BTP engine can beimplemented as a separate library that can be used by other securityproducts/solutions with similar or different requirements.

Referring to FIG. 5, the BTP engine includes a CLIPS Engine 502 thatimplements a RETE algorithm for performing rule matching based on apattern of real-time security events as similarly described above.Monitored events are received as shown at Low Level event 504 (e.g.,various types of events can be monitored, filtered, and aggregated assimilarly described above with respect to FIG. 4). The events arenormalized as described further below with respect to FIG. 6 and thenprovided as CLIPS Fact 506 to be provided as input facts to the CLIPSEngine as shown. The Clips Fact is the data structure format for inputthat can be processed by the Clips Interpreter component of the CLIPSEngine. As similarly described above, the BTP engine of the endpointagent can perform various data pre-processing to support high speedevaluation (e.g., including filtering and normalization of the monitoredreal-time events, such as path Canonization and command line spacesremoval).

The CLIPS Facts are stored as shown at 524 in an Event Store 510 invarious data structures as shown (e.g., (1) By CausalityId; (2) ByTimestamp; etc.). The Event Store can also perform various garbagecollection to remove by Causality (e.g., in response to a CausalityTermination as shown 522), remove by First In First Out (FIFO), and SaveHigh priority facts in the buffer (e.g., facts joining capabilities) toefficiently manages storage space in the Event Store.

The CLIPs Engine can match a set of events to a rule(s) (e.g.,behavioral threat rules) using the RETE algorithm. As similarlydescribed above, these rules can be written in JavaScript and can bestored as a JSON file. The CLIPS Engine can also perform Fact Assertion(Internal fact) determinations as shown at 512. The following is anexample for internal fact that can be inserted into the CLIPS Engine inorder to “enrich” the engine knowledge and provide the ability to writeadditional clips rules based on this internal fact. Specifically, inthis example we are detecting a behavior of a process that started toexecute and generate a new file with the same hash (e.g., which mean theprocess is a copy of itself).

(deftemplate internal.sig_copy_itself_by_hash (slot cid) (slottimestamp) (slot dbg_fact_id)(slot signed)) (defruleinternal.sig_copy_itself_by_hash (and (process_start (cid ?cid)(timestamp ?ps_timestamp) (signer_name ?signer_name) (is_sign ?is_sign)(process_image_path ?ps_process_image_path) (image_path_sha256?ps_image_path_sha256&:(neq ?ps_image_path_sha256 nil)) ;same as infile_operation ) (test (not (is_whitelisted_process?ps_process_image_path ?signer_name))) ;should add 1 more process andcheck who wrote file? (test (not (is_installer_signer?ps_process_image_path ?signer_name ?is_sign))) ) ?the_f <−(file_operation (cid ?cid) (timestamp ?timestamp&:(>= ?timestamp?ps_timestamp)) (sub_type ?fo_sub_type&:(eq ?fo_sub_type ?*file_write*))(file_path_hash ?ps_image_path_sha256) ;same as parent ) => (assert(internal.sig_copy_itself_by_hash (cid ?cid) (timestamp ?timestamp)(dbg_fact_id (fact-index ?the_f)) (signed (and (eq ?is_sign 1)(not(contains ?signer_name “Microsoft”))) ) )) )

As also shown, a Private Heap 508 for Windows-based endpointimplementations can be utilized for improved performance on a MicrosoftWindows® operating system platform. For example, the Microsoft Windows®operating system provides the developer the ability to create a privateheap which means all the memory allocation made with the specific heapID (e.g., called HandleId) will be allocated only on this specificmemory region and not on the default memory heap that each process inthe system has. As such, using this private heap implementation canprovide the following advantages: (1) easier to debug memory issues likememory-leaks, heap-corruptions, etc.; and (2) performance wise—by usinga private memory region, we are limiting the heap fragmentation (e.g., aterm of the OS system) as all the allocations/deallocations are almoston the same sizes (in contrary to the default heap of all the processwhere the allocations are very different as there are many allocatorsfrom many types).

As shown at 514, Installed external functions are also utilized toextend capabilities of the CLIPS Engine (e.g., for REGEX matching,Wildcards matching, etc.). As such, in this example implementation, theopen source/public domain CLIPS Engine is extended to support regularexpression (REGEX) matching. Another example of installed externalfunctions that is utilized in this example implementation to extend theCLIPS Engine for performance improvement includes a fast lookupfunction.

At 516, the CLIPS Engine determines a Rules match with a set of events(e.g., a causality chain). At 520, a SendSecurityEvent API 520 action isperformed in response to the Rule match determination. In this exampleimplementation, in response to the Rules match with a set of events, theCLIPS Engine generates a SecurityEvent which can perform various actionsbased on configuration settings (e.g., generate a pop-up window with aconsole message to the user (on the endpoint), terminate the wholecausality chain (the whole execution chain), generate an XDR UI eventthat presents the CGO (root process of the causality) and some of theinternal behaviors made by the malware, etc.).

In this example implementation, the CLIPS Engine is configurable using aBTP Configuration module 518. For example, the BTP Configuration modulecan be utilized to configure rules (e.g., rule properties), to configurewhitelist values, and/or to configure other module toggles (e.g.,settings and/or parameters, etc.).

Event Filtering Performed by the BTP Engine

In one embodiment, to facilitate an efficient performance of the BTP(e.g., and entire endpoint security agent) in high load eventsenvironments, the agent filters out some of the irrelevant or lesscritical data (e.g., events). As also similarly described above, theevent filtering is configurable/updatable. For example, if a new threatis detected in the wild, then the event filter(s) can be updated suchthat the desired events for detecting a causality chain associated withthe new threat based on a new behavioral threat rule (or ruleadjustment) can be performed by the BTP engine/agent to efficiently andtimely detect this new threat.

In an example implementation, the following is an example for the syntaxthat can be used for the disclosed LRU filtering system (e.g., thissyntax is YAML syntax). In the following example, the agent filters if aLoadImage event is signed by Microsoft and not appear in the ignore list(e.g., internal ignore list as we would like to get some load imagesevent by DLL signed by Microsoft). Another filter scenario is if aprocess is one of the names in the list (e.g., “chrome.exe” just pass itthrough the LRU, which means report it will be reported only once per 24hours).

- provider: image lru: - ttl: *24hours policy_version: min: 8label_bitmap: &ImageLoadLru 1 depth: 30000 attributes: - location:[*AttrCommonCpaPolicy, *AttrCpaCausalityInstanceId] # TODO: can also putAttrCpaActorInstanceId instead - location: [*AttrCommonCpaPolicy,*AttrCpaActorImageName] - location: [*AttrCommonProviderData,*AttrImageFullName] rules: # Drop images that are signed by microsoftexcept a whitelist of images - action: drop policy_version: min: 9conditions: - attribute: location: [*AttrCommonProviderData,*AttrImageSignatureState] type: U64 condition: EQUAL values: - type: INTvalue: *SignatureState_kSigned − attribute: location:[*AttrCommonProviderData, *AttrImageSignatureVendor] type: DATAcondition: EQUAL values: - type: STR values: [“Microsoft Corporation”] −<<: *should_ignore_signed_image # Drop replay images from%systemdrive%\windows - action: drop policy_version: min: 10conditions: - attribute: location: [*AttrCommonProviderData,*AttrImageSignatureState] type: U64 condition: EQUAL values: - type: INTvalue: *SignatureState_kUnsupported − attribute: location:[*AttrCommonIsReplay] type: U64 condition: EQUAL values: - type: INTvalue: 1 − attribute: location: [*AttrCommonProviderData,*AttrImageFullName] type: DATA condition: BEGINS flags: [expand_env]values: - type: STR values: [“%windir %”] − <<:*should_ignore_signed_image - action: id: set-value data: value:*ImageLoadLru location: [*AttrCommonLruLabelBitmap] conditions: -attribute: location: [*AttrCommonCpaPolicy, *AttrCpaActorImageBaseName]type: DATA condition: EQUAL values: - type: STR values: [“chrome.exe”,“firefox.exe”, “iexplore.exe”, “microsoftedge.exe”,“microsoftedgecp.exe”, “slack.exe”]

Whitelisting Performed by the BTP Engine

In one embodiment, the agent (e.g., and the BTP engine) is configuredwith a “trusted signer” whitelist. As also similarly described above,the “trusted signer” whitelist is configurable/updatable.

An example whitelisting configuration is illustrated below.

“Tencent Technology(Shenzhen) Company Limited”, “Baidu Online NetworkTechnology (Beijing) Co., Ltd.”, “GE Intelligent Platforms, Inc.”,“Intel® Software Development Products”, “\”BITT\” LLC″, “1C Company”,“20-20 TECHNOLOGIES INC.”, “20-20 Technologies Inc.”, “2BrightSparks PteLtd”,

“2BrightSparks Pte. Ltd.”,

“2X Software Malta Ltd”,

“360.cn”,

“3D SYSTEMS, INC.”,

Event Normalization Performed by the BTP Engine

FIG. 6 illustrates a process for normalizing an event to a CLIPS Fact inaccordance with some embodiments. In one embodiment, this process fornormalizing events to CLIPS Facts is performed by the BTP engine, suchas similarly described above with respect to FIGS. 4 and 5. As shown inFIG. 6, a Low Level C++ Event 602 is normalized to a CLIPS Fact 604.

Rules for Detecting Malware on an Endpoint Based on Real-Time SystemEvents Using the BTP Engine

In one embodiment, the agent (e.g., and the BTP engine) is configuredwith various rules (e.g., behavior threat rules) for detecting malware(e.g., including new threats) on endpoints. As also similarly describedabove, the rules configurable/updatable as content updates to facilitatemore efficient and dynamic updates for new security threats.

An example malware that can be detected using the disclosed techniquesis a sample of the “hworm” malware family that is detected by thebehaviors it performs (e.g., by using the disclosed BTP engineimplemented techniques, we can change this rule and make it moreaccurate by adding additional behaviors for detecting on the endpointand distribute it to the field via content updates). This example rulealso illustrates an example implementation for an “internal” rule whichrepresents (one of) the behaviors, in this case, this is the script filecreation that is detected by using the Low Level event of file-creation.The script file suffix can be changed anytime and by using thistechnique, we can easily add the new suffix name and catch the newmalware variant. The disclosed BTP implemented techniques alsofacilitate the ability to correlate many events types and behaviors (ontop of it), which thereby can be utilized to generate a complex decisiontree (RETE tree) state that can be efficiently evaluated (e.g., inmicroseconds on a typical endpoint device). This example rule is nowprovided below.

(defrale bioc.hworm “This rule catches behavior that is common in hwormsamples” (internal.external_cscript_or_wscript (cid ?cid))(internal.create_script (cid ?cid)) (internal.autostart_from_local_dir(cid ?cid)) (internal.autostart_by_file_system_change (cid ?cid))(internal.browser_proxy (cid ?cid)) (internal.ie_connection (cid ?cid))(internal.create_doc_exe (cid ?cid)) (internal.start_doc_exe (cid ?cid))(internal.unsigned_process_started (cid ?cid)) (internal.create_exe (cid?cid)) (internal.double_extension (cid ?cid))(internal.double_extension_executable (cid ?cid));(internal.autostart_registry (cid ?cid)) ;internal.autostart (cid?cid)) ;(internal, copy_itself (cid ?cid));(internal.set_hidden_attribute (cid ?cid));(internal.create_hidden_file (cid ?cid));(internal.create_file_on_external_device (cid ?cid)) => ;(bind ?*fires*(+ ?*fires* 1)) ) (defrule internal.create_script ?the_f <−(file_operation (cid ?cid) (timestamp ?timestamp) (sub_type?fo_sub_type&:(eq ?fo_sub_type ?*file_create_new*)) (file_path?fo_file_path&:(is_one_of(file_ext ?fo_file_path) “vbs” “js” “jse” “vbe”“ps1”)) ) (not (internal.create_script (cid ?cid))) => ; FACTS_COUNTERSis 0, no bind ?the_f needed (assert (internal.create_script (cid ?cid)(timestamp ?timestamp) (dbg_fact_id (fact-index ?the_f)))) )

Another example rule for an event which is converted to a Clips Fact isthe allocate_virtual_memory_remote, which is illustrated below.

(deftemplate allocate_rirtual_memory_remote (slot cid) ;stringBase64(slot pid) ;int (slot instance_id) ;stringBase64 (slottarget_instance_id) ;String (slot tid) ;int (slot timestamp) ;long (slotis_injected) ;0 or 1 (slot target_pid) ; String (slot base_address);String (slot region_size) ; String (slot alloc_type) ; String (slotprot_mask) ; String (slot event_id) ;stringBase64 (slot dse_internal);nil #ifdef is_impersonated (slot is_impersonated) #endif#ifdefimage_tracker (slot base_address_mapped_image_path) ; String -get_map_image_name on thread start_address (slotbase_address_suspicious_symbol) ; String - get_map_image_name on threadstart_address #endif#ifdef windows #ifdef actor (slot actor_pid) (slotactor_instance_id) (slot actor_tid) (slot actor_thread_instance_id)#endif #endif )

Yet another example rule is a rule for the bioc.file_link_exploit, whichis based on a Windows exploit. In this example, once the exploit wasdiscovered by our security analysts/research group, a new rule that cancatch the malicious behavior was generated (e.g., in this example theattacker can take advantage of the file link feature and use it toexploit the WER (Windows Error Report) system) and after testing it, wedeployed it in the wild using a Content update. This example rule isillustrated below.

(defrule bioc.file_link_exploit (process_start) (cid ?cid) (instance_id?instance_id) (integrity_level ?integrity_level&: (is_one_of?integrity_level ?*integrity_low* ?*integrity_medium* ) ) ) ?the_f <−(file_operation (instance_id ?instance_id) (sub_type?fo_subtype&:(is_one_of ?fo_subtype ?*file_link* ?*file_sym_link*))(old_file_path ?fo_link_src) (canonized_old_file_path?fo_canonized_link_src&: (not (starts_with_one_of ?fo_canonized_link_src“%programfiles” “%systemdrive” “%system32” “%windir” ) ) )(canonized_file_path ?fo_canonized_link_target&: (starts_with_one_of?fo_canonized_link_target “%programfiles” “%systemdrive” “%system32”“%windir” ) ) (file_path ?fo_link_target&: (is_one_of (file_ext?fo_link_target) “exe” “dll” “sys” “ini” “js” “psl”) ) (timestamp?fo_link_timestamp) ) ; Any access by a high-priv process using the FSlink created by the low-priv process (process_start (cid ?cid_priv)(instance_id ?instance_id_priv) (integrity_level ?integrity_level_priv&:(is_one_of ?integrity_level _priv ?*integrity_high* ?*integrity_system*) ) ) (file_operation (cid ?cid_priv) (instance_id ?instance_id_priv);(sub_type ?sub_type&:(is_one_of ?sub_type ?*file_remove*?*file_write*)) (file_path ?fo_link_src) (timestamp ?timestamp_priv&:(>=?timestamp_priv ?fo_link_timestamp)) ) (not (bioc.file_link_exploit (cid?cid))) => (assert (bioc.file_link_exploit (cid ?cid) (timestamp?fo_link_timestamp) (dbg_fact_id (fact- index ?the_f)))) )

Behavioral Threat Protection Analysis from the Security Agent Console

FIG. 7 is a screen diagram illustrating a behavioral threat protectionanalysis from a security agent console of an endpoint agent inaccordance with some embodiments. In one embodiment, the security agentconsole shown in FIG. 7 is a component of endpoint security agents 154,156, 158, and 160.

As shown in this example in the screen shot 702 of the security endpointconsole in FIG. 7, the disclosed Behavioral Threat Protection (BTP)engine detects and performs an action(s) in response to detect malwareattack/threat activity by monitoring for malicious sequences/chains ofevents across processes, network activities, file system activities,etc. (e.g., and can be configured to terminate and/or report on suchmalware attacks/threats when detected). As shown in this example, whenthe security endpoint detects these malicious events, the securityendpoint console presents a timeline of actions taken in an eventanalysis tab in the management console.

Example Processes Performed by Malware Analysis System for DetectingMalicious Activity on an Endpoint Based on Real-Time System Events

FIG. 8 is a flow diagram illustrating a process for detecting maliciousactivity on an endpoint based on real-time events in accordance withsome embodiments. In one embodiment, process 800 is performed using thesystem architectures described above (e.g., such as described above withrespect to FIGS. 1-7).

At 802, monitoring an endpoint for malicious activity using an endpointagent is performed. For example, the endpoint can be a local device(e.g., a mobile device, laptop, desktop computer, IoT device, etc.).

At 804, detecting malicious activity associated with an application onthe endpoint based on real-time system events using the endpoint agentbased on a set of rules is performed.

At 806, in response to detecting malicious activity on the endpointbased on real-time system events using the endpoint agent, performing asecurity response based on a security policy is performed.

FIG. 9 is another flow diagram illustrating a process for detectingmalicious activity on an endpoint based on real-time events inaccordance with some embodiments. In one embodiment, process 900 isperformed using the system architectures described above (e.g., such asdescribed above with respect to FIGS. 1-7).

At 902, a content update that includes a set of security rules isreceived. As similarly described above, the rules (e.g., behavior threatrules) can be efficiently distributed to endpoint agents as contentupdates.

At 904, the set of security rules is compiled. For example, the rulescan be compiled into a RETE tree for efficient pattern matching usingRETE techniques as similarly described above.

At 906, monitoring an endpoint for malicious activity using an endpointagent is performed. For example, the endpoint can be a local device(e.g., a mobile device, laptop, desktop computer, IoT device, etc.).

At 908, detecting malicious activity associated with an application onthe endpoint based on real-time system events using the endpoint agentbased on the compiled set of rules is performed. As similarly describedabove, in response to detecting malicious activity on the endpoint basedon real-time system events using the endpoint agent, a security responsebased on a security policy can be performed.

FIG. 10 is another flow diagram illustrating a process for detectingmalicious activity on an endpoint based on real-time events inaccordance with some embodiments. In one embodiment, process 1000 isperformed using the system architectures described above (e.g., such asdescribed above with respect to FIGS. 1-7).

At 1002, monitoring real-time events on an endpoint using an endpointagent is performed. For example, the endpoint can be a local device(e.g., a mobile device, laptop, desktop computer, IoT device, etc.).

At 1004, the real-time events are aggregated and filtered on theendpoint. For example, the events can be filtered to reduce the noiseassociated with innocuous and/or less critical events, and the filterscan be configurable via content updates as similarly described above.

At 1006, the aggregated and filtered real-time events are normalized.For example, the events can be normalized for efficient processing usingthe CLIPS Engine as similarly described above.

At 1008, detecting malicious activity associated with an application onthe endpoint based on real-time events using the endpoint agent based ona set of rules is performed. For example, various behavior threat rulescan be configured to detect a pattern of events (e.g., causality chains)associated with malware and/or new threats as similarly described above.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A system, comprising: a processor configured to:monitor an endpoint for malicious activity using an endpoint agent,wherein the endpoint comprises a local device; detect malicious activityassociated with an application on the endpoint based on real-time systemevents using the endpoint agent based on a set of rules; in response todetecting malicious activity on the endpoint based on real-time systemevents using the endpoint agent, perform a security response based on asecurity to policy; and a memory coupled to the processor and configuredto provide the processor with instructions.
 2. The system of claim 1wherein the processor is further configured to detect an attempt by theapplication to take an action that would violate the set of rules,wherein the set of rules is includes one or more updated detectionrules.
 3. The system of claim 1, wherein the processor is furtherconfigured to detect an attempt by the application to take an actionthat would violate the set of rules, and wherein the set of rulescomprises a whitelisted set of behaviors observed at a remote serverduring emulation of a sample in a virtualized environment and wherein anattempt by the application while executing on the local device to takean action not included in the whitelisted set of behaviors constitutes arule violation.
 4. The system of claim 1, wherein the processor isfurther configured to: detect an attempt by the application to take anaction that would violate the set of rules, and report the attempt to auser of the endpoint.
 5. The system of claim 1, wherein the processor isfurther configured to: detect an attempt by the application to take anaction that would violate the set of rules, and report the attempt to aremote server.
 6. The system of claim 1, wherein the processor isfurther configured to report the detected malicious activity to a remoteserver, wherein in response to receiving the report, the remote serverperforms an evaluation of a sample provided by the endpoint, wherein thesample is associated with the detected malicious activity.
 7. The systemof claim 1, wherein the set of rules restrict processes associated witha sample to behaviors observed during an execution of the sample in avirtualized environment.
 8. The system of claim 1, wherein a remoteserver is configured to evaluate an updated version of the applicationin response to receiving an indication that the application has beenupdated.
 9. The system of claim 1, wherein a remote server is configuredto evaluate the application at least in part by executing theapplication in a virtualized environment, and wherein endpoint agent isconfigured to implement, at the endpoint, a set of rules restrictingbehaviors of an application.
 10. A method, comprising: monitoring anendpoint for malicious activity using an endpoint agent, wherein theendpoint comprises a local device; detecting malicious activityassociated with an application on the endpoint based on real-time systemevents using the endpoint agent based on a set of rules; and in responseto detecting malicious activity on the endpoint based on real-timesystem events using the endpoint agent, performing a security responsebased on a security policy.
 11. The method of claim 10 furthercomprising detecting an attempt by the application to take an actionthat would violate the set of rules, wherein the set of rules includesone or more updated detection rules.
 12. The method of claim 10 furthercomprising detecting an attempt by the application to take an actionthat would violate the set of rules, and wherein the set of rulescomprises a whitelisted set of behaviors observed at a remote serverduring emulation of a sample in a virtualized environment and wherein anattempt by the application while executing on the local device to takean action not included in the whitelisted set of behaviors constitutes arule violation.
 13. The method of claim 10 further comprising: detectingan attempt by the application to take an action that would violate theset of rules; and reporting the attempt to a user of the endpoint. 14.The method of claim 10 further comprising: detect an attempt by theapplication to take an action that would violate the set of rules, andreport the attempt to a remote server.
 15. The method of claim 10further comprising reporting the detected malicious activity to a remoteserver, wherein in response to receiving the report, the remote serverperforms an evaluation of a sample provided by the endpoint, wherein thesample is associated with the detected malicious activity.
 16. Themethod of claim 10, wherein the set of rules restrict processesassociated with a sample to behaviors observed during an execution ofthe sample in a virtualized environment.
 17. The method of claim 10wherein a remote server is configured to evaluate an updated version ofthe application in response to receiving an indication that theapplication has been updated.
 18. The method of claim 10, wherein aremote server is configured to evaluate the application at least in partby executing the application in a virtualized environment, and whereinendpoint agent is configured to implement, at the endpoint, a set ofrules restricting behaviors of an application.
 19. A computer programproduct, the computer program product being embodied in a tangiblecomputer readable storage medium and comprising computer instructionsfor: monitoring an endpoint for malicious activity using an endpointagent, wherein the endpoint comprises a local device; detectingmalicious activity associated with an application on the endpoint basedon real-time system events using the endpoint agent based on a set ofrules; and in response to detecting malicious activity on the endpointbased on real-time system events using the endpoint agent, performing asecurity response based on a security policy.
 20. The computer programproduct recited in claim 19, further comprising computer instructionsfor: detecting an attempt by the application to take an action thatwould violate the set of rules, wherein the set of rules includes one ormore updated detection rules.